Compute node cooling with air fed through backplane

ABSTRACT

A computing system includes a chassis, one or more backplanes coupled to the chassis. Computing devices are coupled to the one or more backplanes. The one or more backplanes include backplane openings that allow air to pass from one side of the backplane to the other side of the backplane. Air channels are formed by adjacent circuit board assemblies of the computing devices and the one or more backplanes. Channel capping elements at least partially close the air channels.

BACKGROUND

Organizations such as on-line retailers, Internet service providers,search providers, financial institutions, universities, and othercomputing-intensive organizations often conduct computer operations fromlarge scale computing facilities. Such computing facilities house andaccommodate a large amount of server, network, and computer equipment toprocess, store, and exchange data as needed to carry out anorganization's operations. Typically, a computer room of a computingfacility includes many server racks. Each server rack, in turn, includesmany servers and associated computer equipment.

Computer systems typically include a number of components that generatewaste heat. Such components include printed circuit boards, mass storagedevices, power supplies, and processors. For example, some computerswith multiple processors may generate 250 watts of waste heat. Someknown computer systems include a plurality of such larger,multiple-processor computers that are configured into rack-mountedcomponents, and then are subsequently positioned within a rack system.

Some known data centers include methods and apparatus that facilitatewaste heat removal from rack systems. Many existing methods andapparatus may not, however, provide air for cooling in an effectivemanner to where it is most needed. Moreover, some known data centersinclude rack systems having configurations that are non-uniform withrespect to component density and usage, such that different parts of arack generate waste heat at a different rate compared to other parts ofthe rack.

In many rack systems, fans are used to move air through rack-mountedcomputer systems to remove heat from components in the computer systems.For example, air may be moved from front to back in the rack. Typically,some areas within a given computer system receive more cooling air thanothers. Components in areas of a computer system that are not wellcooled may be prone to failure.

As with other components, servers fail from time to time while inservice. To restore the systems to full operation, servers may need tobe powered down and removed from a rack so that the defective componentscan be replaced or repaired. In systems where many servers are mountedon a common chassis, all of the servers on the chassis must be withdrawnfrom the rack in order to service one failed server. In this case,cooling air to all of the servers on that chassis is lost, which mayimpair performance or cause additional failures in the servers on thechassis.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates one embodiment of a computing system having computenodes with channel-capping elements mounted on a common backplane withair flow introduced into the channels from under a backplane.

FIG. 2 illustrates one embodiment of a rack housing a computing systemwith sled-mounted compute nodes having channel-capping elements.

FIG. 3 illustrates one embodiment a computing system with sled-mountedcompute nodes partially withdrawn from a rack.

FIG. 4 is a top view illustrating one embodiment of a computing systemhaving two rows of compute nodes on two separate backplanes.

FIG. 5 is a side view illustrating one embodiment of a computing systemwith multiple rows of compute nodes having inter-node channels linked toone another.

FIG. 6 is a top view illustrating one embodiment of a computing systemreconfigured to include three rows of compute nodes on three separatebackplanes.

FIG. 7 illustrates a connector element between that can coupleinter-node channels on adjacent rows in a computing system.

FIGS. 8A and 8B illustrate top and side views of a computing system withsled-mounted compute nodes installed in a rack.

FIGS. 9A and 9B illustrate top and side views of a computing system withsled-mounted compute nodes with the sled partially withdrawn from therack with continued air flow through channels between compute nodes.

FIGS. 10A and 10B illustrate top and side views of a computing systemwith one compute node removed and continued air flow through channelsbetween other remaining compute nodes.

FIGS. 11A and 11B illustrate top and side views of a computing systemwith a dummy plate in covering a gap where a compute node has beenremoved.

FIG. 12 illustrates one embodiment of a computing system with statusindicator lights for compute nodes mounted on a backplane.

FIG. 13 illustrates a front view of a capping element that enclosesmultiple channels between compute nodes.

FIG. 14 illustrates one embodiment of a rack-mountable shelf unit withcompute nodes mounted on backplanes.

FIG. 15 illustrates one embodiment of a compute node mounted onbackplane assembly.

FIG. 16 illustrates a partially exploded view of one embodiment of acompute node with a channel-capping plate.

FIG. 17 illustrates one embodiment of a chassis for compute shelf with abackplane mounting plate.

FIG. 18 illustrates one embodiment of a chassis for a compute node shelfwith cross-braces that partition rows of compute nodes.

FIG. 19 is a side view of a compute shelf with arrows indicating airflow through the shelf.

FIG. 20 illustrates cooling computing devices mounted on one or morebackplanes with flow through channels between circuit board assemblies.

FIG. 21 illustrates altering a configuration of computing system thatincludes removing a set of computing devices in rows at one pitch, andreplacing them with a set of computing devices in rows having adifferent pitch.

FIG. 22 illustrates cooling computing devices includes moving air frombelow a backplane into channel such that the air flows down the lengthof the channel.

FIG. 23 illustrates compute node maintenance with continuous cooling.

FIG. 24 illustrates a rear view of one embodiment of a rack systemhaving rear-mounted fans.

While the invention is susceptible to various modifications andalternative forms, specific embodiments thereof are shown by way ofexample in the drawings and will herein be described in detail. Itshould be understood, however, that the drawings and detaileddescription thereto are not intended to limit the invention to theparticular form disclosed, but on the contrary, the intention is tocover all modifications, equivalents and alternatives falling within thespirit and scope of the present invention as defined by the appendedclaims. The headings used herein are for organizational purposes onlyand are not meant to be used to limit the scope of the description orthe claims. As used throughout this application, the word “may” is usedin a permissive sense (i.e., meaning having the potential to), ratherthan the mandatory sense (i.e., meaning must). Similarly, the words“include,” “including,” and “includes” mean including, but not limitedto.

DETAILED DESCRIPTION OF EMBODIMENTS

Various embodiments of computer systems, and systems and methods forperforming computing operations, are disclosed. According to oneembodiment, a computing system includes a chassis, an air inlet end, anair exit end, one or more backplanes coupled to the chassis. Computingdevices are coupled to the one or more backplanes. The computing devicesmay include a circuit board assembly and one or more processors. Thecomputing system includes an air passage adjacent to the backplanes andair channels. The air channels can channel air between pairs of adjacentcircuit board assemblies. The air passage receives air from the airinlet end of the system. The one or more backplanes include openingsthat allow air to pass from the air passage to the air channels. One ormore of the openings is farther from the air inlet end than at least oneother of the openings.

According to one embodiment, a method of cooling computing devices on achassis includes introducing air at a first end of a system into an airpassages on one side (for example, the bottom side) of one or morebackplanes. Air from air passage is moved through openings in thebackplanes and in channels formed between adjacent circuit boardassemblies mounted on the other side (for example, the top) of thebackplanes. Air is expelled at a second end of the system that isopposite the first end. One or more of the openings is farther from theair inlet end than at least one other of the openings.

According to one embodiment, a method of changing a configuration ofcomputer system includes removing a first set of backplanes from achassis for the computing system to create open space in the chassis. Asecond set of backplanes is installed the open space in the chassis. Thesecond set of backplanes supports a second set of computing devices intwo or more rows between one end and another end of the chassis at apitch from row to row that is different from that of the first set ofcomputing devices.

According to one embodiment, a computing system includes a chassis, oneor more backplanes coupled to the chassis. Computing devices are coupledto the one or more backplanes. The one or more backplanes includebackplane openings that allow air to pass from one side of the backplaneto the other side of the backplane. Air channels are formed by adjacentcircuit board assemblies of the computing devices and the one or morebackplanes. Channel capping elements at least partially close the airchannels.

According to one embodiment, a method of cooling computing devicesincludes moving air below a backplane through one or more openings inthe backplane into one or more air channels formed by adjacent circuitboard assemblies on the backplane. Air is moved down the air channels toremove heat from heat producing components on the circuit boardassemblies.

According to one embodiment, a system includes a rack and computingsystems coupled to the rack. The computing systems may include achassis, computing devices coupled to the chassis, and air channelsformed between at least two of the computing devices. The rack supportsa computing system while the computing system is partially withdrawnfrom the rack. The computing system may remain in operation while thecomputing system is partially withdrawn from the rack. Air may bedirected through the air channels while the computing system ispartially withdrawn from the rack and remains in operation.

According to one embodiment, a method of performing maintenance on acomputing system includes at least partially withdrawing a computingsystem from a rack, moving air through air channels between adjacentcircuit boards in the computing system while the computing system is inthe withdrawn position. Operations to maintain the computing system areperformed while the computing system is in the withdrawn position andwhile air is moved through the air channels between circuit boards.

As used herein, “air handling system” means a system that provides ormoves air to, or removes air from, one or more systems or components.

As used herein, “air moving device” includes any device, element,system, or combination thereof that can move air. Examples of air movingdevices include fans, blowers, and compressed air systems.

As used herein, an “aisle” means a space next to one or more elements,devices, or racks.

As used herein, “backplane” means a plate or board to which otherelectronic components, such as mass storage devices, circuit boards, canbe mounted. In some embodiments, hard disk drives are plugged into abackplane in a generally perpendicular orientation relative to the faceof the backplane. In some embodiments, a backplane includes and one ormore power buses that can transmit power to components on the backplane,and one or more data buses that can transmit data to and from componentsinstalled on the backplane.

As used herein, “ambient” means, with respect to a system or facility,the air surrounding at least a portion of the system or facility. Forexample, with respect to a data center, ambient air may be air outsidethe data center, for example, at or near an intake hood of an airhandling system for the data center.

As used herein, a “cable” includes any cable, conduit, or line thatcarries one or more conductors and that is flexible over at least aportion of its length. A cable may include a connector portion, such asa plug, at one or more of its ends.

As used herein, “circuit board” means any board or plate that has one ormore electrical conductors transmitting power, data, or signals fromcomponents on or coupled to the circuit board to other components on theboard or to external components. In certain embodiments, a circuit boardis an epoxy glass board with one or more conductive layers therein. Acircuit board may, however, be made of any suitable combination ofmaterials.

As used herein, “chassis” means a structure or element that supportsanother element or to which other elements can be mounted. A chassis mayhave any shape or construction, including a frame, a sheet, a plate, abox, a channel, or a combination thereof. In one embodiment, a chassisis made from one or more sheet metal parts. A chassis for a computersystem may support circuit board assemblies, power supply units, datastorage devices, fans, cables, and other components of the computersystem.

As used herein, to “close” an open channel means to at least partiallyenclose the channel such that air is inhibited from escaping thechannel.

As used herein, “computing” includes any operations that can beperformed by a computer, such as computation, data storage, dataretrieval, or communications.

As used herein, “computing systems” includes any of various systems ordevices in which computing operations can be carried out, such ascomputer systems or components thereof. One example of a computingsystem is a rack-mounted server that includes one or more computingdevices. As used herein, the term computing device is not limited tojust those integrated circuits referred to in the art as a computer, butbroadly refers to a processor, a server, a microcontroller, amicrocomputer, a programmable logic controller (PLC), an applicationspecific integrated circuit, and other programmable circuits, and theseterms are used interchangeably herein. Some examples of computingdevices include e-commerce servers, network devices, telecommunicationsequipment, medical equipment, electrical power management and controldevices, and professional audio equipment (digital, analog, orcombinations thereof). In various embodiments, memory may include, butis not limited to, a computer-readable medium, such as a random accessmemory (RAM). Alternatively, a compact disc-read only memory (CD-ROM), amagneto-optical disk (MOD), and/or a digital versatile disc (DVD) mayalso be used. Also, additional input channels may include computerperipherals associated with an operator interface such as a mouse and akeyboard. Alternatively, other computer peripherals may also be usedthat may include, for example, a scanner. Furthermore, in the someembodiments, additional output channels may include an operatorinterface monitor and/or a printer.

As used herein, “data center” includes any facility or portion of afacility in which computer operations are carried out. A data center mayinclude servers dedicated to specific functions or serving multiplefunctions. Examples of computer operations include informationprocessing, communications, testing, simulations, power distribution andcontrol, and operational control.

As used herein, to “direct” air includes directing or channeling air,such as to a region or point in space. In various embodiments, airmovement for directing air may be induced by creating a high pressureregion, a low pressure region, or a combination both. For example, airmay be directed downwardly within a chassis by creating a low pressureregion at the bottom of the chassis. In some embodiments, air isdirected using vanes, panels, plates, baffles, pipes or other structuralelements.

As used herein, “member” includes a single element or a combination oftwo or more elements (for example, a member can include two or moresheet metal parts fastened to one another.

As used herein, a “module” is a component or a combination of componentsphysically coupled to one another. A module may include functionalelements and systems, such as computer systems, circuit boards, racks,blowers, ducts, and power distribution units, as well as structuralelements, such a base, frame, housing, or container.

As used herein, “open channel” means a channel that is open along atleast a portion of the length of the channel.

As used herein, “primarily horizontal” means more horizontal thanvertical. In the context of an installed element or device, “primarilyhorizontal” includes an element or device whose installed width isgreater than its installed height.

As used herein, “primarily vertical” means more vertical thanhorizontal. In the context of an installed element or device, “primarilyvertical” includes an element or device whose installed height isgreater than its installed width. In the context of a hard disk drive,“primarily vertical” includes a hard disk drive that is installed suchthat the installed height of the hard disk drive is greater than theinstalled width of the hard disk drive.

As used herein, a “rack” means a rack, container, frame, or otherelement or combination of elements that can contain or physicallysupport one or more computer systems.

As used herein, “room” means a room or a space of a building. As usedherein, “computer room” means a room of a building in which computersystems, such as rack-mounted servers, are operated.

As used herein, a “space” means a space, area or volume.

As used herein, “shelf” means any element or combination of elements onwhich an object can be rested. A shelf may include, for example, aplate, a sheet, a tray, a disc, a block, a grid, or a box. A shelf maybe rectangular, square, round, or another shape. In some embodiments, ashelf may be one or more rails.

As used herein, “shock absorbing”, as applied to a supporting elementfor another element, means that the supporting element absorbsmechanical energy and/or dampens shock and/or vibration loads. Ashock-absorbing material may be elastic, viscoelastic, viscous, orcombinations thereof.

In some embodiments, a computing system (for example, a computingmodule) includes a chassis, one or more backplanes in a horizontalorientation, and computing devices (for example, compute nodes) on thebackplanes. The computing system has an air inlet end and an air exitend. The computing system includes an air passage under the backplanesand air channels between the computing devices on the backplanes.Openings in the backplanes allow air to move from the air passage underthe backplanes into the air channels between the computing devices. Someof the openings in the backplanes are farther away from the system airinlet end than other openings (for example, deeper in the chassis). Theair passage under the backplanes and the air channels may run lengthwisebetween the system air inlet and the system air exit (for example,lengthwise front-to-back). In some cases, the chassis accommodatesmultiple backplanes arranged in a row from front to back of the chassis(for example, one backplane behind another). The number of backplanesand depth of each backplane may be changed depending on the needs of thesystem.

FIG. 1 illustrates one embodiment of a computing system having computenodes mounted on a common backplane with air flow introduced into thechannels from under a backplane. System 100 includes rack 102 andcomputing system 104. Computing system 104 is mounted on rails 106 ofrack 102. Rails 106 may be fixed on vertical posts in rack 102 (notshown for clarity).

Computing system 104 includes sled 108, risers 110, backplane 112, andcompute node assemblies 114. Sled 108 may serve as a chassis forcomponents of computing system 104. Each of compute node assemblies 114includes compute node printed circuit board 116, processors 118, andchannel-capping rim 120.

Backplane 112 includes backplane circuit board 121, card guides 122 andopenings 124. Each of compute node assemblies 114 is mounted onbackplane 112. Backplane openings 124 are provided in backplane circuitboard 121. Backplane openings 124 are spaced from one another in a rowfrom front to rear of backplane 112.

Channels 126 are formed between adjacent compute node assemblies 114.Each channel 126 is formed by a pair of adjacent compute node printedcircuit boards 116 and the portion of backplane circuit board 121between those two adjacent compute node printed circuit boards 116.Openings 124 may be located on circuit board 121 in backplane circuitboard 121 between each pair of adjacent compute node printed circuitboards 116. Each of compute node assemblies 114 may include network I/Oconnectors, power connectors, and signal connectors to support operationof compute node assemblies 114.

Computing system 104 may include one or more power supply units. Eachpower supply unit may supply power to one or more of compute nodeassemblies 114. In one embodiment, a power supply unit is installed atthe front of a computing system (for example, in front of the row ofcompute node printed circuit boards 116).

Computing system 104 may include one or more mass storage devices, suchas hard disk drives. Each mass storage device may be accessible by oneor more of compute node assemblies 114. In certain embodiments, massstorage devices are mounted on a common chassis with printed circuitboards for one or more compute nodes.

Shared resource module 117 is mounted at the front of sled 108. Sharedresource module 117 may include modules to provide resources to computenodes 130. Compute node assemblies 114 may share common resourcedevices, such as power supply units, mass storage devices (for example,hard disk drives) or network switches. Shared resource module 114 mayinclude an enclosure or mounting brackets. Shared resource module 117may block air from entering in the gaps between compute node assemblycircuit boards 116, such that all of the air entering the system isintroduced to the inter-node channels through openings in backplane 112.

Backplane 112 is amounted on sled 108 by way of risers 110. Risers 110create space 128 between sled 108 and backplane 112. Front inlet opening130 to space 124 is at the front of computing system 104 between thefront edges of sled 108 and backplane 112. A back plate may be includedat the rear edges of sled 108 and backplane 112 to cap off space 128 atthe rear of the sled.

Sled 108 may be carried on rails 106 of rack 102. Computing system 104may be slid in and out of rack 102 on sled 108. In various embodiments,computing system 104 may be supported on other types of support systems,such as a shelf, plate, hangers, or telescoping rails.

In various embodiments described above, each of the processors on acircuit board may operate as a separate compute node. In certainembodiments, however, circuit board assemblies on a dual-processor boardmay cooperate to function as a single compute node. In certainembodiments, two or more processors on a multiple processor circuitboard assembly share access to some or all of the hard disk drives in acompute module.

Channel-capping rims 120 may serve as channel-capping and/orchannel-enclosing elements for channels 126. In the embodimentillustrated in FIG. 1, channel-capping rims 120 may be in the form of anangle attached to compute node printed circuit board 116 near the top ofthe circuit board. In various embodiments, however, a channel betweencircuit boards may be closed by other elements or members. Examples ofother closing elements include plates, blocks, rims, beads, or plugs ofvarious dimensions.

In some embodiments, channels between adjacent circuit boards are fullyclosed. In certain embodiments, the edge of channel-capping rim 120attached to one circuit board may seal on the adjacent circuit board. Insome embodiments, a channel-enclosing element includes a sealingelement. For example, each of channel-capping rims 120 may include arubber seal along its edge that contacts the adjacent circuit board toenclose channels 126.

In some embodiments, an air moving device creates negative pressure atthe rear of computing system 104. Air may be drawn through front inletopening 130 and into space 128 under backplane 112. Air in space 128 maybe drawn through backplane openings 124 and into channels 126. Air inchannels 126 may flow upwardly and toward the rear of computing system104 and across heat producing components on compute node printed circuitboards 116. Channel-capping elements 120 may contain the moving air inchannels 126. Air in space 124 may remain unheated until it passes intochannels 126 and receives heat from heat producing components on thecircuit boards. As such, air entering channels 126 through backplaneopenings 124 near the rear of computing system 104 may be as cool as airentering channels 126 through backplane openings 124 near the rear ofcomputing system 104.

For illustrative purposes, rack 102 is shown with only one of computingsystems 104. In various embodiments, however, a rack may hold any numberof computing systems, compute nodes, systems or components.

FIG. 2 illustrates one embodiment of a rack housing a computing systemwith sled-mounted compute nodes having channel-capping elements. System140 includes rack 142 and computing system 144. Computing system 144 maybe similar to computing system 104 described above relative to FIG. 1.Rack 142 includes computing system shelf 146. Shelf 146 includes box 148and rear opening 150. Computing system 144 is supported on shelf 146.Rack 142 includes rear fans 152. Rear fans 152 may draw air throughfront inlet opening 130 into space below backplane 112, throughbackplane openings 124 in backplane 112 and into channels 126, throughchannels 126, and through rear opening 150 of shelf 146. Air movingthrough channels 126 may remove heat from heat producing components oncompute node assemblies 114 of computing system 144.

For illustrative purposes, rack 142 contains only one of computingsystems 144. In various embodiments, however, a rack may hold any numberof computing systems, compute nodes, systems or components.

In some embodiments, a computing system includes a chassis, one or morebackplanes, and computing devices with circuit boards mounted in one ormore rows on the backplanes. The computing system includes an airpassage under the backplanes and air channels between the circuit boardson the backplanes. Openings in the backplanes allow air to move from theair passage under the backplanes into the air channels between thecomputing devices. Channel-capping elements (for example, plates) atleast partially close the air channels between the computing devices.Each channel-capping element may close an air channel between a pair ofadjacent circuit boards. The closed air channel may contain an airstream that cools components on the circuit boards. The channel-cappingelements can be attached to each of the circuit board assemblies.

In some embodiments, a system includes a rack, a computing system, andone or more air moving devices. The rack can support the computingsystem in a withdrawn position (for example, slid out on rails from thefully installed position). The computing system remains in operationwhile the computing system is in the withdrawn position from the rack.The computing system includes computing devices (for example, computenodes on circuit board assemblies). Air can be moved through airchannels between the computing devices while the computing system is inthe withdrawn position, such that the computing devices can continue tobe cooled while the computing system is being serviced (for example, torepair or replace compute nodes). In some cases, the compute nodes aremounted on one or more backplanes. Air for cooling the compute nodes maybe introduced through openings in the backplanes.

FIG. 3 illustrates one embodiment of a computing system withsled-mounted compute nodes partially withdrawn from a rack. In theembodiment shown in FIG. 3, sled 108 has been withdrawn on shelf 146,but remains supported on shelf 146 in the withdrawn position. In someembodiments, some or all of the compute nodes of computing system 144remain in operation while computing system 144 is in the withdrawnposition.

In some embodiments, system 140 includes a cable management device. Thecable management device may carry cables that connect computing system144 to systems external to the computing system. In one embodiment, thecable management device includes an articulated arm (for example, a“zee” linkage with hinged connections between the links of the arm).Sections or links of the arm may unfold or unfurl when the computingdevice is pulled out from the fully installed position in the rack.

During operation of computing system 144 in the withdrawn position, rearfans 152 may continue to draw air through computing system 144. Inparticular, rear fans 152 may draw air through front inlet opening 130into space below backplane 112, through openings in backplane 112 andinto channels 126, and to the rear in channels 126. Box 148 may duct airexpelled from channels 126 through rear opening 150 of shelf 146. Rearfans 152 may move the heated air out of the rack. In this manner,airflow may be maintained while computing system 144 is the partiallywithdrawn position. Airflow across components of computing system 144may be maintained, for example, during maintenance of compute nodes ofcomputing system 144.

In some embodiments, a computing system includes multiple rows ofcompute nodes. Each row of compute nodes may be on a differentbackplane. FIG. 4 is a top view illustrating one embodiment of acomputing system having two rows of compute nodes on separatebackplanes. FIG. 5 is a side view illustrating one embodiment of thecomputing system shown in FIG. 4. Computing system 160 includes sled162, risers 164, backplanes 166, and compute nodes assemblies 168.Channels 170 are formed between adjacent pairs of compute nodeassemblies 168.

Air for cooling compute node assemblies 168 may be drawn into channels170 through openings in the backplanes 166. In some embodiments, air ispulled from front to back of channels in the direction shown by thearrows in FIGS. 4 and 5. The arrangement of air moving devices andmanner of operation may be similar to that above relative to FIGS. 2 and3. Channel capping rims 172 may contain air in channels 170. Bridgeplate 172 may duct air from the first row of compute node assemblies tothe second row of compute node assemblies on the second backplane.

In some embodiments, a computing system may be reconfigured to increaseor reduce the number of rows compute nodes in the system. In someembodiments, the number of backplanes included from front to back of thecomputing system is increased or decrease. For example, referring toFIGS. 4 and 5, the number of backplanes/rows of compute nodes may beincreased from two to three. FIG. 6 is a top view illustrating oneembodiment of a computing system reconfigured to include three rows ofcompute nodes on three separate backplanes. Reconfigured system 180includes three rows of computing devices 182. Each row is installed on aseparate backplane 164. Relative to FIG. 4, the spacing betweenbackplanes has been adjusted to accommodate the number of rows ofcompute nodes to be installed. In some embodiments, the same backplaneis used regardless of the number of rows in the configuration. Forexample, the same backplane may be used in the three-row configurationshown in FIG. 6 as the two-row configuration shown in FIG. 4.

In some embodiments, elements are included to maintain continuity ofairflow from one channel to another. FIG. 7 illustrates a connectorelement that may be used to couple inter-node channels on adjacent rowsin a computing system. In FIG. 7, connector element 200 connects achannel formed between circuit boards on row 204 with a channel formedbetween circuit boards on row 208.

FIGS. 8A, 8B, 9A, 9B, 10A, 10B, 11A, and 11B illustrate operation andmaintenance of a computing system with airflow maintained duringmaintenance operations on the computing system. FIGS. 8A and 8Billustrate top and side views of a computing system with sled-mountedcompute nodes installed in a rack. System 240 includes rack 242 andcomputing system 244. Computing system 244 may be similar to computingsystem 160 described above relative to FIG. 4. Rack 242 includescomputing system shelf 246. Shelf 246 includes box 248 and rear opening250. Computing system 244 is supported on shelf 246. Rack 242 includesrear fans 252. Rear fans 252 may draw air through front inlet opening130 into space below backplane 112, through openings in backplane 112and into channels 126, through channels 126, and through rear opening250 of shelf 246. Air moving through channels 126 may remove heat fromheat producing components on compute node assemblies 114 of computingsystem 244.

Computing system 244 includes dividers 247 between rows of computenodes. Dividers 247 may serve as barriers between adjacent rows.Dividers 247 may inhibit air flow between adjacent rows up to the heightof dividers 247. Above the top edge of dividers 247, air may flow freelybetween rows. Accordingly, the air heated by each row of compute nodesmay be drawn rearward and expelled from the rear of 242.

FIGS. 9A and 9B illustrate top and side views of a computing system withsled-mounted compute nodes with the sled partially withdrawn from therack with continued air flow through channels between compute nodes.Sled 108 has been withdrawn on shelf 246, but remains supported on shelf246 in the withdrawn position. With sled 108 in the withdrawn position,compute node assemblies 254 are accessible to allow service personnel toperform maintenance on computing system 244. In some embodiments, someor all of compute nodes of 254 computing system 244 remain in operationwhile computing system 244 is in the withdrawn position.

FIGS. 10A and 10B illustrate top and side views of a computing systemwith one compute node removed and continued air flow through channelsbetween other remaining compute nodes. Compute node 254 a has beenremoved from backplane 112. The removal of compute node 254 a creates agap in the channel-capping elements at the top of the array of circuitboards. With compute node 254 a removed, channel 126 a is no longerenclosed to duct air between the adjacent circuit boards, such that aleak or breach is created in the closed channel airflow arrangement. Assuch, air flow through backplane openings 124 a may be substantiallyreduced. In addition, to the extent air is drawn through backplaneopenings 124 a, much of the air may escape out the top opening, asillustrated by flow arrows 258 a. Nevertheless, in all of the otherchannels 126 of computing system 244, air may continue to flow throughthe channels from front to back, such that cooling is maintained for allof the compute node assemblies adjacent to those channels.

In some embodiments, filler elements may be provided to fill enclosechannels between circuit boards of a computing system. Filler elementsmay take the form of dummy modules, filler plates, bars, strips, andlids. FIGS. 11A and 11B illustrate top and side views of a computingsystem with a filler plate covering a gap where a compute node has beenremoved. Filler plate 260 is installed at the location of the opening atthe top of channel 126 a. Filler plate 260 may close channel 126 a suchthat flow is restored in channel 126 a, similar to that of the otherchannels 126 between the circuit boards on computing system 244.

In the system described above relative to FIGS. 11A and 11B, fillerelements described in the context of maintaining airflow throughchannels between compute nodes during maintenance of one or more of thenodes. Filler elements may, however in various embodiments be used inany part of the life cycle of a computing system. For example, dummymodules or filler plates may be installed at open positions of acomputing system when the computing system is initially built. Thefiller elements may be replaced by operating modules (for example, ifadditional compute capacity is needed for the system).

In some embodiments, compute nodes on a computing system include statusindicators. The status indicators may include, for example, one or morelight emitting diode indicators on each circuit board in an array ofcircuit boards mounted on backplanes on a common sled. FIG. 12illustrates one embodiment of a computing system with status indicatorlights for compute nodes mounted on a backplane. System 300 includescompute node assemblies 302. Compute node assemblies 302 include statusindicators lights 304 and channel-capping rims 306. Status indicatorlights 306 are mounted on compute node circuit boards 308.Channel-capping rims 306 may be transparent or translucent such thatlight from status indicator lights 304 on compute node circuit boards308 shines through the channel-capping rims 306. Examples of materialsthat may be used for channel capping rims 306 include polycarbonate andpoly methyl methacrylate (“PMMA”). In certain embodiments, statusindicator lights for a compute node may be mounted on top of the computenode (for example, on a channel-capping rim attached to the compute nodecircuit board).

Status indicator lights 304 on compute node assemblies 302 may provideinformation about the status of the compute node. For example, a redstatus indicator light may indicate that the compute node has failed,while a green status light indicator may indicate that the compute nodeis working properly. Status light indicators may provide a visual aid tomaintenance personnel to assess which of the compute nodes needsservice.

In various embodiments described above, the channel-capping element capsa single channel, and is attached to one of the circuit boards adjacentto a channel. Channel-enclosing elements may nevertheless in variousembodiments enclose more than one channel. In addition,channel-enclosing elements may be attached to two or more circuitboards, or be attached to structural elements other than the circuitboards. FIG. 13 illustrates a front view of a capping element thatencloses multiple channels between compute nodes. System 320 includessled 322, backplane 324, compute node assemblies 326, andchannel-enclosing cover 328. Channel enclosing cover 328 may be attachedto two or more of compute node assemblies 326. Channel enclosing cover328 may enclose channels 330 between compute node assemblies 326.

FIG. 14 illustrates one embodiment of a rack-mountable shelf unit withcompute nodes mounted on backplanes. Compute shelf 400 includes computenode modules 402, shared resource module 404, chassis 406, cross-braces408, and channel capping plates 410. Compute shelf 400 may be mountablein a rack. Compute node modules 402 may be mounted on chassis 406 by wayof one or more backplanes. Shared resource module 404 may be mountedchassis 406. (Some of the locations where channel capping plates couldbe installed are shown without plates for illustrative purposes).

Shared resource module 404 may include modules to provide resources tocompute nodes 402. For example, shared resource module 404 includespower supply units 412. Compute nodes 402 may share other commonresources, such as mass storage devices (for example, hard disk drives)or network switches.

Cross-braces 408 may subdivide the row of compute node modules 402. Insome embodiments, cross braces hold the side edges of the circuit boardsof compute node modules 402. The lower portion of cross braces 408 mayserve as a barrier that inhibits cooling air from passing between thedifferent rows of compute node modules 402. Cross-braces 408 may includeair flow openings 414. Air flow openings 414 may allow air to flow fromone row to another near the top of shelf 404. Channel capping plates 410may inhibit air flowing in the channels from dissipating out the top ofchassis 406.

FIG. 15 illustrates one embodiment of a compute node mounted onbackplane assembly. Backplane assembly 418 includes backplane printedcircuit board 420 and card guides 421. A compute node module 402 may bemounted in each of card guides 421. Backplane printed circuit board 420includes apertures 422. Apertures 422 allow airflow through backplaneprinted circuit board 420.

FIG. 16 illustrates a partially exploded view of one embodiment of acompute node with a channel-capping plate. Compute node module 402includes mounting plate 423, compute node printed circuit board 424,processor assembly 425, memory modules 426, and indicator lights 427.Processor assembly 425 includes a processor and heat sink mounted tocompute node printed circuit board 424. Mounting plate 423 includes tabs428. Channel capping plate 410 may be attached to tabs 428. In someembodiments, channel capping plate 410 is clear or translucent, suchthat the state of indicator lights 427 (on/off, color) can be seen fromabove the shelf by maintenance personnel.

FIG. 17 illustrates one embodiment of a chassis for compute shelf with abackplane mounting plate. Chassis 406 include chassis enclosure 429 andbottom duct member 430. Bottom duct member 430 is mounted at the bottomof chassis enclosure 429. Bottom duct member 430 includes apertures 431.Apertures 431 allow air to flow from duct 433 (for example, throughbackplane assemblies mounted on the duct).

FIG. 18 illustrates one embodiment of a chassis for a compute node shelfwith cross-braces that partition rows of compute nodes. Cross-braces 408are mounted at spaced intervals from front to back in chassis 406.Cross-braces 408 may serves as guide or supports for compute nodemodules mounted on backplanes between adjacent cross braces. Wall 432may serve as a barrier that keeps air flowing within a given row ofcompute nodes (rather than, for example, dissipating into rows fartherback in the shelf.

FIG. 19 is a side view of a compute shelf with arrows indicating airflow through the shelf. An air moving device, such as a fan, may moveair in through bottom inlet 440 and into lower duct space 441. Lowerduct space 441 may serve as a plenum for air to be fed into channelsbetween compute nodes. Air may pass from lower duct space 441 throughapertures 431 in bottom duct member 430, through apertures in backplaneassembly 418, and upwardly across heat producing components on computenode modules 402, as shown by the arrows in FIGS. 14 and 19. Once theheated air reaches the top part of the channels, the air may be movedout of the shelf via upper channel 444 (successively, through air flowopenings 414) toward the rear of compute shelf 400 (as shown, forexample, by arrows in FIGS. 15 and 19.) Channel capping plates 410 maycontain the heated air within upper channel 444. At the rear of theshelf, air may be expelled from upper channel through exhaust vent 446.

In various embodiments described above, a compute node circuit boardsare mounted in a vertical orientation on top of a backplane mounted in ahorizontal orientation. Compute nodes and backplanes withcross-backplane airflow may nevertheless in various embodiments bemounted in any orientation. In one embodiment, compute nodes are mountedhorizontally on a backplane that is in a vertical orientation. Air mayflow through a duct on the side of the backplane that is opposite to thecompute nodes, and then pass through the backplane to feed channelsbetween compute nodes.

In some embodiments, compute nodes are mounted in a vertical orientationto the underside of a backplane that is in a horizontal orientation. Anair passage above the backplane may feed air through into channelsbetween the compute nodes.

FIG. 20 illustrates cooling computing devices mounted on one or morebackplanes with flow through channels between circuit board assemblies.At 450, air at a first end of a system is introduced into one or morepassages adjacent to one or more backplanes. In some embodiments, thepassage is below the backplane and the circuit board assemblies areabove the backplanes.

At 452, air from one or more air passages is moved through openings inthe backplanes. One or more of the openings may be farther from thefirst end than at least one other of the openings.

At 454, air is moved in channels formed between adjacent circuit boardassemblies mounted on the backplanes. At 456, air in the channels isexpelled at a second end of the system.

In some embodiments, the channels are enclosed to contain a stream ofair flowing in the channels. In one embodiment, a channel is enclosed byattaching capping elements (for example, channel-capping rims 120described above relative to FIG. 1) to one or more of the circuit boardassemblies adjacent to the channel.

FIG. 21 illustrates altering a configuration of computing system thatincludes removing a set of computing devices in a set of rows at onepitch, and replacing them with a set of computing devices in a set ofrows having a second pitch. At 460, a first set of backplanes is removedfrom a chassis for the computing system to create open space in thechassis. The first set of backplanes may support computing devices intwo or more rows between one end and another end of the chassis at afirst pitch from row to row.

At 462, a second set of backplanes is installed in the open space in thechassis. The second set of backplanes supports a second set of computingdevices in two or more rows between one end and another end of thechassis at a second pitch from row to row, the second pitch from row torow of the second set of computing devices is less than or greater thanthe first pitch from row to row of the first set of computing devices.For example, the first set of computing devices may be arranged in threerows, each of the three rows being mounted on a separate backplane,while the replacement set of computing devices is mounted in two rows,each of the two rows being mounted on separate backplane. The number ofcomputing devices, the number of rows of computing devices, or both, maybe changed during reconfiguration of the computing system.

FIG. 22 illustrates cooling computing devices includes moving air frombelow a backplane into channel such that the air flows down the lengthof the channel. At 470, air is moved below a backplane through openingsin the backplane into one or more air channels formed by adjacentcircuit board assemblies on the backplane.

At 472, air is moved down the air channels to remove heat from heatproducing components on the circuit board assemblies. In someembodiments, the channels are capped to inhibit air from leaving thechannel. For example, a channel-capping rim may be attached to a circuitboard to close an air channel on either or both sides of the circuitboard. In some embodiments, air is moved successively through channelsin two or more rows of computing devices, such as shown above relativeto FIGS. 4 through 6.

FIG. 23 illustrates compute node maintenance with continuous cooling. At480, a computing system is partially withdrawn from a rack. For example,the computing system may be pulled out on rails of the rack such thatthe computing system remains supported on the rails.

At 482, air is moved through air channels between adjacent circuitboards in the computing system while the computing system is in awithdrawn position.

At 484, operations are performed to maintain the computing system whilethe computing system is in the withdrawn position and while air is movedthrough the air channels between circuit boards. In some embodiments, acomputing device is hot-swapped while other computing devices continueto be cooled by air flowing in air channels between adjacent circuitboards. In some embodiments, air is moved from below a backplane onwhich the computing devices in the rack are mounted.

In some embodiments, backplanes are mounted to reduce or minimizetransmission of shock and/or vibration loads between each hard diskdrive and a chassis and between hard disk drives within a system. Forexample, backplane circuit board assemblies may be mounted on pads. Thepads may be made of a shock absorbing material, such as an elastomericmaterial. The pads may reduce transmission of shock and/or vibrationbetween a shelf or chassis and the compute node circuit boards of acomputing system.

In some embodiments, elements of disk drive backplanes and a chassis maycombine to form a box section mounting for hard disk drives. Forexample, the shelf, spacers, and one or more of the backplane circuitboard assemblies illustrated in FIG. 1 may combine to form a rectangularbox section. The box section may reduce deformation of a chassis, suchas sagging of chassis bottom panel. In some embodiments, rails, pads, atray, or similar structural elements may serve multiple functions,including forming the box section structure, space for cable runs, andspace for air flow.

In some embodiments, the size and number of opening in a backplane maybe selected to tune the air flow through various compute nodes achassis. For example, in one embodiment, the vents for the backplanesnear the rear of the chassis may larger than the vents for thebackplanes near the front of the chassis, since a greater airflow may berequired near the rear of the chassis because of the relatively warm airin that portion of the chassis.

In some embodiments, a rack-level air moving devices are implemented byway of a fan door. FIG. 24 illustrates a rear view of one embodiment ofa rack system. System 500 includes rack 502 and rear door 504. Rear door504 couples with rack 502 on hinges 506. Fan modules 508 couple with,and are supported by, rear door 504.

In some embodiments, fan modules 508 include alternating current (AC)fans. In one embodiment, the fans have an input voltage rating of about100V-120 V. In one embodiment, the fans have an input voltage rating ofabout 230 V. Fan modules 300 may receive power from rack level powerdistribution units. In some embodiments, fan modules 508 in a rack arehot swappable. In some embodiments, a manual power switch is providedfor each of fan modules 508.

In one embodiment, each fan operates at a flow rate between 50 to 100cubic feet per minute. In one embodiment, each of the fans operates at aflow rate about 200 cubic feet per minute.

In some embodiments, a system may include variable speed fans. Incertain embodiments, power switching and/or fan speed may be controlledautomatically. Fans may be controlled individually, or in groups of twoor more fans. In some embodiments, fans are controlled based on sensorsdata (for example, temperature sensors in the rack).

In some embodiments, one or more fans for a rack system may becontrolled via a control system. In certain embodiments, a controlsystem includes at least one programmable logic controller. The PLC mayreceive measurements of conditions in the rack or at other locations ina data center. A PLC may receive data corresponding to air flow rate,temperature, pressure, humidity, or various other operating orenvironmental conditions.

In one embodiment, the PLC receives data from one or air flow sensorsthat measure airflow in the rack. Based on sensor data, the PLC maycontrol parameters such as fan speed, as appropriate for the prevailingoperational conditions. In another embodiment, the PLC receives datafrom one or more temperature sensors that measure temperature in therack and/or at other locations in a data center. In certain embodiments,a PLC may modulate dampers between open and closed positions to modulateairflow, as appropriate for the prevailing operational conditions.

In some embodiments, rack-mounted computing modules are commonly cooledby a cooling air system that delivers air to the rack. To remove heatfrom computing modules installed in the rack, an air handling system maybe operated to cause air to flow in computer room and through the racksystem. As the air reaches the front of each of computing modules, theair may pass through the chassis of the computing modules. After passingthrough the chassis, the heated air may exit the rear of the rack systemand flow out of the computer room. In certain embodiments, computingmodules may have on board fans in addition to, or lieu of, a centralcooling system. In certain embodiments, a rack may have a fan thatsupplies cooling air to all of the computing modules in the rack.

For clarity, modules or other components in many of the figures hereinhave been shown with a simple box outline around functional components.In various embodiments, a module or a chassis for a module may includean enclosure, a tray, a mounting plate, a combination thereof, as wellas various other structural elements.

Although the embodiments above have been described in considerabledetail, numerous variations and modifications will become apparent tothose skilled in the art once the above disclosure is fully appreciated.It is intended that the following claims be interpreted to embrace allsuch variations and modifications.

What is claimed is:
 1. A computing system, comprising: a chassis; an airinlet end; an air exit end opposite the air inlet end; one or morebackplanes coupled to the chassis; a plurality of computing devicescoupled to the at least one backplane, wherein at least two of thecomputing devices comprise a circuit board assembly and one or moreprocessors coupled to the circuit board assembly; one or more airpassages adjacent to at least one of the backplanes, wherein at leastone of the air passages is configured receive air from the air inlet endof the system; and one or more air channels, wherein at least one of theair channels is configured to channel air between two or more adjacentcircuit board assemblies of the computing devices on the at least onebackplane, wherein at least one of the air channels is configuredchannel air toward the air exit end and expel air at the air exit end ofthe system, wherein the at least one backplane comprises a plurality ofopenings configured to allow air to pass from at least one of the airpassages to at least one of the air channels, wherein at least one ofthe openings is farther from the air inlet end than at least one otherof the openings.
 2. The computing system of claim 1, wherein air atleast one of the air channels runs lengthwise between the air inlet endand the air exit end of the system.
 3. The computing system of claim 1,wherein in at least one of the channels is configured to channel airupwardly across at least one of the circuit board assemblies and atleast one of the channels is configured to channel air toward the airexit end of the system.
 4. The computing system of claim 1, wherein theone or more backplanes comprise two or more backplanes coupled to thechassis, wherein at least one row of computing devices is coupled toeach of the two or more backplanes.
 5. The computing system of claim 1,wherein the one or more backplanes comprise two or more backplanescoupled to the chassis, wherein at least two of the backplanes arearranged successively from the air inlet end to the air exit end.
 6. Thecomputing system of claim 1, further comprising one or more air movingdevices configured to move air through at least one of the one or moreair passages under the backplanes.
 7. The computing system of claim 1,further comprising one or more channel-capping elements, wherein atleast one of the channel-capping elements is configured to at leastpartially close at least one of the channels.
 8. The computing system ofclaim 1, further comprising one or more air moving devices configured tomove air from one or more passages on one side of the at least onebackplane through the one or more openings into at least one of thechannels.
 9. The computing system of claim 1, wherein the computingsystem is mounted in a rack, wherein the computing system is configuredto remain in operation when the computing system is at least partiallywithdrawn from the rack, wherein at least one of the air channels isconfigured to channel air from below the at least one backplane when thecomputing system is at least partially withdrawn from the rack.
 10. Thecomputing system of claim 1, wherein the at least one backplane is in aprimarily horizontal orientation, wherein the at least one backplane isconfigured to allow air to pass from under the at least one backplaneinto one or more air channels between adjacent computing devices. 11.The computing system of claim 1, wherein the plurality of computingdevices comprise two or more rows of computing devices.
 12. Thecomputing system of claim 11, wherein the computing system furthercomprises one or more barriers between at least two of the rows, whereinat least one of the barriers is configured to inhibit air from movingbetween adjacent rows of the computing devices.
 13. The computing systemof claim 1, further comprising one or more shared resource devicescoupled to the chassis, wherein at least one of the shared resourcedevices is configured to provide shared resources to two or more ofcomputing devices.
 14. A method of cooling computing devices on achassis, comprising: introducing air at a first end of a system into oneor more passages on a first side of one or more backplanes; moving atleast a portion of the air from one or more air passages through aplurality of openings in at least one of the backplanes, wherein atleast one of the openings is farther from the first end than at leastone other of the openings; and moving air in one or more channels formedbetween two or more adjacent circuit board assemblies mounted on a sideof the at least one backplane that is opposite the first side; andexpelling, at a second end of the system that is opposite the first end,at least a portion the air from at least one of the channels.
 15. Themethod of claim 14, further comprising at least partially enclosing atleast one of the channels to at least partially contain a stream of airflowing in the at least one channel.
 16. The method of claim 14, whereinat least partially enclosing at least one of the channels comprisesattaching one or more capping elements to one or more of the circuitboard assemblies.
 17. The method of claim 14, wherein moving air it oneor more channels comprises moving air in a channel formed in each of twoor more successive rows of computing devices arranged between the firstend of the computing system and the second end.
 18. The method of claim14, wherein introducing air comprises: introducing a first portion ofair through openings on a first backplane supporting a first row ofcomputing devices; and introducing a second portion of air throughopenings on a second backplane supporting a second row of computingdevices.